{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Building Data Pipeline With DataLoader\n",
    "This tutorial shows how to loading data with DataLoader, a powerful tool to handle multiple dataset.\n",
    "- `Dataset` and `Loader` introduction\n",
    "- Construct data iterator "
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "`Dataset` is an object to search needed data in one or more given directories.\n",
    "It has 3 APIs for globing files.\n",
    "\n",
    "- `Dataset.include(*args)` glob standard glob pattern\n",
    "- `Dataset.include_reg(*args)` and `Dataset.exclude(*args)` glob regular expression\n",
    "\n",
    "`Loader` is an object construct using `Dataset`.\n",
    "\n",
    "If `Loader` is constructed only from `hr_data`, it generates **LR** data bicubically.\n",
    "If `Loader` is constructed from both `hr_data` and `lr_data`, it pairs two dataset.\n",
    "\n",
    "The data iterator is got by `Loader.make_one_shot_iterator(...)`, which can be called multiple times.\n",
    "Each item is a `dict` object containing `hr`, `lr` and `name` entries.\n",
    "The shape of LR data is specified by `batch_shape` (hence the shape of HR is `batch_shape * scale`).\n",
    "\n",
    "`batch_shape` also supports 5-D for videos such as [4, 3, 3, 16, 16] (N, T, C, H, W).\n",
    "For the channel placement, the `DATA_FORMAT` comes with the backend:\n",
    "- For **pytorch**, it's `\"channels_first\"`\n",
    "- For **tensorflow**, it's `\"channels_last\"`."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "outputs": [
    {
     "name": "stdout",
     "text": [
      "(4, 3, 64, 64) (4, 3, 16, 16)\n",
      "['img_005_SRF_2_LR', 'c_10', 'img1', 'img_005_SRF_2_LR']\n",
      "(4, 3, 64, 64) (4, 3, 16, 16)\n",
      "['img0', 'xiuxian003', 'xiuxian003', 'img_001_SRF_2_LR']\n",
      "(4, 1, 16, 16) (4, 1, 16, 16)\n",
      "['xiuxian003', 'xiuxian001', 'xiuxian003', 'xiuxian003']\n",
      "(4, 1, 16, 16) (4, 1, 16, 16)\n",
      "['xiuxian002', 'xiuxian001', 'xiuxian001', 'xiuxian001']\n"
     ],
     "output_type": "stream"
    }
   ],
   "source": [
    "from VSR.DataLoader import Dataset, Loader\n",
    "\n",
    "data1 = Dataset('../../Tests/data').include('*.png')  # glob all png files under /mnt/data\n",
    "data2 = Dataset('../../Tests/data')  # glob all supported files under data2\n",
    "data2_hr = data2.include_reg('\\\\bhr\\\\b')\n",
    "data2_lr = data2.include_reg('\\\\blr\\\\b')\n",
    "\n",
    "loader1 = Loader(hr_data=data1, scale=4, threads=1)\n",
    "loader2 = Loader(hr_data=data2_hr, lr_data=data2_lr)\n",
    "\n",
    "itr1 = loader1.make_one_shot_iterator(batch_shape=[4, 3, 16, 16], steps=2, shuffle=True)\n",
    "for item in itr1:\n",
    "    label = item['hr']\n",
    "    feature = item['lr']\n",
    "    print(label.shape, feature.shape)\n",
    "    print(item['name'])\n",
    "\n",
    "loader2.set_color_space('lr', 'L')\n",
    "loader2.set_color_space('hr', 'L')\n",
    "itr2 = loader2.make_one_shot_iterator(batch_shape=[4, 1, 16, 16], steps=2, shuffle=True)\n",
    "for item in itr2:\n",
    "    label = item['hr']\n",
    "    feature = item['lr']\n",
    "    print(label.shape, feature.shape)\n",
    "    print(item['name'])"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n",
     "is_executing": false
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "- [x] Next: [Calling Model with Executor](3.%20Calling%20model%20with%20executor.ipynb)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  },
  "pycharm": {
   "stem_cell": {
    "cell_type": "raw",
    "source": [],
    "metadata": {
     "collapsed": false
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}